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1  PROJECT  SUMMARY 


Wicab,  Inc.  has  developed  novel  technology  which  allows  information  from  external  devices 
to  be  sensed  by  humans  via  neuro-stimulation  of  the  tongue.  Applications  are  numerous  and 
include  the  BrainPort®  balance  device  to  assist  patients  with  vestibular  deficits,  the  BrainPort 
vision  device,  a  sensory  augmentation  device  for  the  blind,  and  the  BrainPort  Underwater 
Sensory  Substitution  System  which  provides  navigational  cues  to  military  divers.  Current 
devices  present  data  to  the  tongue  using  low  density  electrode  arrays  (100-625  elements, 
~1  in2).  Present  and  future  applications  may  benefit  from  array  densities  and  pulse  timing 
patterns  that  match  the  spatial  resolution  of  the  tongue.  Therefore,  characterization  of  this 
resolution  is  the  key  to  full  exploitation  of  the  tongue  sensory  channel. 

This  program  represents  one  year’s  research  effort  to  empirically  measuring  the  electro¬ 
stimulation  properties  of  the  tongue.  Advanced  hardware  and  software  were  developed  to 
support  experimental  protocols.  Under  the  current  program  funding,  to  understand  tongue 
acuity  limits  with  respect  to  electro-stimulation,  approximately  60  subjects  and  more  than  200 
subject  hours  were  involved  in  the  following  experiments: 

•  Two-point  discrimination 

•  Temporal  gap  detection 

•  Tongue  mapping 

•  Spatial  summation 

•  Contrast  sensitivity 

The  goal  of  this  project  was  to  provide  empirical  evidence  concerning  the  psychophysical 
limits  of  the  tongue.  While  the  results  from  these  experiments  begin  to  characterize  the 
information  channel,  they  do  not  represent  the  true  limits  of  the  channel.  In  order  to  assess 
the  absolute  limit,  four  experimental  components  need  to  be  optimized:  subjects,  training, 
application,  and  technology. 

Our  results,  with  ~15  subjects  per  experiment,  demonstrate  that  there  is  individual  variation  in 
task  performance.  Our  participants  were  not  chosen  for  their  tongue  perception  ability,  so  our 
results  may  miss  “super-performers”  which  would  improve  the  threshold  limit.  In  addition, 
subjects  did  not  receive  extensive  practice  with  the  stimuli,  and  thus  practice  could  push  the 
psychophysical  limits.  Second,  we  found  that  the  task  demands  affects  the  reported 
threshold.  For  example,  two  point  discrimination  was  ~0.75mm  but  spatial  gratings  threshold 
was  closer  to  ~0.5mm.  Therefore,  to  define  the  absolute  threshold,  either  the  task  needs  to 
be  fixed  to  a  task  of  interest  or  there  needs  to  be  converging  evidence  from  a  variety  of 
experiments  testing  similar  skills.  Finally,  subject  performance  is  only  as  precise  as  the 
equipment  being  used. 

To  report  a  definite  threshold  we  need  to  have  hardware  that  exceeds  performance  in  order 
to  narrow  in  on  the  exact  limit.  In  conclusion,  while  the  suite  of  studies  provides  empirical 
data  regarding  the  psychophysical  characteristics  of  the  tongue,  they  should  not  be 
interpreted  as  the  absolute  limits. 
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1.1  Significance 


•  Electro-tactile  psychophysical  characteristics  of  the  tongue 

Two  Point  Spatial  Discrimination  is  on  average  0.75mm,  with  some  individuals  able  to 
detect  0.25mm  or  better.  This  implies  that  arrays  on  the  order  of  0.125mm  center- 
center  electrode  spacing  (or  better)  may  be  useful  for  high  performing  individuals. 

Temporal  Discrimination  is  on  average  75ms,  with  some  individuals  performing  at 
50ms  or  better  gap  detection.  This  implies  that  array  frame  rates  of  20  Hz  or  faster 
should  be  useful  for  high  performing  individuals. 

Test  data  suggests  that  tongue  sensitivity  is  a  function  of  stimulation  location  as  well 
as  number  of  stimulating  tactors.  In  addition,  the  dynamic  range  in  voltage  (stimulation 
intensity)  is  a  function  of  the  number  of  stimulating  tactors.  This  data  may  have 
implications  on  the  electro-mechanical  design  of  high  density  arrays,  as  well  as  the 
stimulation  waveform  and  power  requirements. 

Preliminary  data  from  static  2-D  grating  tests  indicates  that  average  spatial 
discrimination  is  0.6mm  (compared  to  0.75mm  for  2pt  tests).  This  implies  that  there 
may  be  strategies  to  further  improve  the  average  spatial  discrimination  ability  and  is 
the  subject  of  continuing  effort. 

Spatial  acuity  and  temporal  acuity  appear  to  differ  within  individuals  -  high  performers 
in  one  modality  are  not  necessarily  high  performers  in  another.  This  presents 
opportunities  to  train  individuals  whose  physiology  allows  peak  performance  across 
modalities,  to  tailor  information  to  individual  performance,  and  to  tailor  arrays  for 
specific  applications. 

Finally,  data  from  this  study  confirms  that  electro-neurostimulation  of  tongue  is  on  par 
with  other  stimulation  modalities:  mechanical  grating  ~0. 500mm  (Van  Boven,  1994) 
and  vibro-tactile  at  ~20Hz  (Ezawa,  1988). 

*  High  Density  Array  Design  and  Experiment  Control 

Experimental  data  indicates  0.125mm  electrode  spacing  (200/inch)  should  match  the 
tongue’s  spatial  acuity.  Using  a  simple  fabrication  technique,  arrays  with  0.152mm 
spacing  were  assembled  (densities  beyond  this  level  requires  a  significant  technology 
investment).  These  near-optimal  arrays  allow  advanced  testing  of  dynamic 
information  on  the  tongue. 

An  electrical  control  system  was  designed  and  implemented  which  allows  concurrent 
control  of  all  electrodes  in  an  array.  This  system  manages  the  safe  delivery  of 
stimulation  to  any  number  of  simultaneous  electrodes,  as  well  as  manipulation  of  the 
return  path  geometry.  In  addition,  stimulation  waveform  patterns  can  be  defined  for 
evaluation  of  waveform  on  spatial  and  temporal  detection  properties.  The  system  is 
designed  to  manage  up  to  27,000  electrodes,  although  for  practical  testing,  we  have 
limited  the  current  implementation  to  2000  element  arrays. 
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An  experimental  control  software  application  was  developed  to  allow  researchers  to 
define  experiments  using  any  configuration  of  electrode  geometry  and  stimulation 
waveform  patterns.  Experiment  sessions  are  defined  and  executed  to  present 
stimulus  information  to  subjects  and  record  their  responses.  All  data  is  archived  for 
off-line  analysis. 

All  human  subject  testing  was  done  under  an  IRB  approved  protocol. 

Results  of  experiments  and  analysis  of  data  are  presented  below,  followed  by  a  summary  of 
significant  hardware  and  software  development. 


2  STUDIES  AND  RESULTS 
2.1  Two-Point  Discrimination 

Two-point  discrimination  was  performed  using  a  modified  BrainPort  Balance  Device  C200 
(BBD-C200).  Nine  linear  arrays  with  ten  electrodes  each  were  produced  with  industry 
standard  printed  wiring  board  technology.  The  individual  electrodes  ranged  in  diameter  from 
1 ,5mm  to  0. 1 69mm  and  were  spaced  (center  to  center)  at  a  distance  of  1 .5  times  the 
electrode  diameter.  The  smallest  electrode  size  and  spacing  represents  the  highest 
resolution  readily  available  from  commercial  printed  wiring  board  vendors.  Sufficient  arrays 
were  produced  so  that  each  test  subject  used  a  new  array,  minimizing  the  effects  of 
electrode  corrosion  and  resultant  changes  in  impedance. 

The  stimulus  generation  circuit  employed  in  the  BBD-C200  consists  of  a  ColdFire  32-bit 
microcontroller,  Texas  Instruments  digital  to  analog  converter  (TLV5630),  Burr-Brown 
operational  amplifier  (OPA4132),  and  a  bank  of  Analog  Devices  demultiplexers 
(ADG408BRU)  used  for  routing  the  stimulus  signal  to  any  one  of  one  hundred  electrode 
addresses.  Each  output  of  the  demultiplexers  are  connected  to  an  AC  coupling  capacitor 
(0.1  pF  ceramic)  with  the  opposite  side  of  that  capacitor  connected  to  the  tongue  placed 
electrode. 

Internal  pilot  studies  suggested  that  some  subjects  could  discriminate  two  stimuli  spaced 
0.5mm  apart.  As  a  result,  we  selected  a  linear  electrode  array  with  10  electrodes,  each  169 
micrometers  (p)  diameter  and  spaced  254p  apart  (center  to  center)  for  the  formal  two  point 
discrimination  experiment.  This  array  enabled  conditions  that  could  both  exceed 
performance  and  also  be  within  the  range  of  successful  performance.  Each  active  electrode 
presented  a  continuously  repeating  stimulation  waveform.  The  waveform  scheme  consisted 
of  six  double  stimulating  pulses  (21 ,4psec  each  separated  by  5psec)  repeating  at  369Hz. 
After  every  6  pulses,  2  pulses  were  presented  at  0V  (total  period  of  waveform  was  21 ,68ms). 
When  two  tactors  were  presented,  the  first  tactor  fired  51 ,6psec  after  the  second  tactor. 
Pulse  amplitude  was  fixed  per  subject.  Each  subject  set  their  own  comfortable  working 
amplitude  up  to  24.5V  prior  to  the  experiment,  providing  comfortable  stimulation.  This 
waveform  scheme  was  selected  because  sensation  was  perceived  as  a  comfortable  and 
continuous  stimulation. 
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Sixteen  subjects  (10  men;  6  women)  participated,  ranging  in  age  from  18-39  years  (mean 
24.5yrs).  Experiment  duration  was  approximately  two  hours.  Subjects  were  recruited  from 
the  Madison,  Wl  area.  All  subjects  gave  informed  consent  with  a  study  protocol  approved  by 
the  New  England  IRB.  Pulse  amplitude  (in  volts)  was  determined  individually  before  running 
the  two  point  discrimination  experiment.  Subjects  were  presented  with  trials  containing  one 
or  two  tactors  firing  on  an  array.  They  were  instructed  to  manipulate  a  hand  held  slider 
control,  allowing  each  subject  to  explore  and  adjust  the  stimulation  to  a  comfortable  working 
level.  Once  a  comfortable  level  was  reached,  subjects  indicated  their  response  by  pressing  a 
button  on  the  hand-held  control.  “Working  level”  was  defined  as  a  stimulation  level  that  was 
strong  and  comfortable  for  at  least  five  seconds.  Intensity  values  (voltage)  were  gathered  for 
one  and  two  contiguous  firing  tactors.  Each  measurement  was  repeated  several  times 
across  multiple  tactor  locations.  For  each  subject,  the  mean  “working  level”  for  one  and  two 
contiguous  points  was  used  as  the  fixed  voltage  intensity  for  the  two  point  discrimination  test. 
These  values  ranged  from  18-25V  across  participants  (mean  21.46V). 

Trials  were  presented  in  five  experimental  runs,  blocked  by  the  separation  between  two 
tactors.  These  conditions  were:  0.254mm  (no  gap),  0.580mm  (1  tactor  gap),  0.762mm  (2 
tactor  gap),  1 .01 6mm  (3  tactor  gap)  and  1 .261  mm  (4  tactor  gap).  In  each  block,  half  of  the 
trials  contained  only  one  stimulating  tactor,  while  the  other  half  contained  two  firing  tactors  of 
the  fixed  condition  distance.  The  order  of  the  five  experimental  blocks  was  pseudo- 
randomized  for  each  subject. 

Subjects  were  instructed  to  press  one  button  on  the  hand-held  control  if  they  felt  one  stimulus 
on  their  tongue  and  a  different  button  if  they  felt  two  stimuli  on  their  tongue.  They  were 
encouraged  to  use  the  tip  of  their  tongue  (or  any  other  part  of  the  tongue)  to  search  for 
and/or  to  explore  the  stimulation  in  order  to  respond  maximally.  There  was  no  response  time 
limit.  In  this  way,  subjects  could  explore  the  stimulus  with  whatever  part  of  the  tongue  they 
felt  gave  them  the  best  information. 

To  familiarize  subjects  with  electrotactile  stimulation  and  the  procedures,  subjects  were  given 
5-10  minutes  of  interactive  practice  trials.  Participants  sampled  a  few  trials  of  each  of  the 
conditions,  becoming  acquainted  with  the  trial  types.  During  this  practice  phase,  the 
researcher  labeled  the  conditions  as  being  truly  two  points  or  one.  Subjects  were  told  there 
may  be  trials  where  they  may  not  be  able  to  confidently  give  a  correct  response.  They  were 
reassured  that  this  was  necessary  in  order  to  get  to  the  limit  of  spatial  resolution  on  the 
tongue.  During  experimental  testing,  performance  feedback  was  not  provided. 

The  primary  endpoint  will  be  a  threshold  defined  by  a  group  d-prime  greater  than  or  equal  to 
one  (by  subject  and  by  group),  indicating  subjects  are  sensitive  to  the  presence  of  two 
categories  (CBASEE  1985).  Calculating  d-prime  (d’),  a  measure  of  sensitivity  in  a 
discrimination  task,  is  computed  by  the  standardized  difference  between  the  false  alarm  rate 
and  the  hit  rate:  d’=z(H)-z(FA)  (Macmillan  &  Creelman  2005).  The  hit  rate  is  the  proportion  of 
correctly  identified  trials  of  two  points,  when  two  points  were  actually  stimulated.  The  false 
alarm  rate  is  the  proportion  of  incorrect  trials  where  the  subject  reported  two  points,  but  only 
one  point  was  actually  stimulated.  The  numeric  output  of  d-prime  represents  the  perceived 
distance  between  the  two  categories.  A  d-prime  near  zero  represents  chance  discrimination, 
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where  subjects  are  insensitive  to  the  condition  categories.  When  d-prime  is  greater  than 
one,  at  least  one  standard  deviation  separates  the  two  response  categories  indicating  that 
subjects  are  sensitive  to  and  correctly  responding  to  the  condition  categories. 

As  a  secondary  endpoint,  individual  d-prime  measures  by  condition  were  statistically 
explored  post-hoc  using  a  one  way  ANOVA.  Individual  differences  were  further  explored  by 
computing  the  percentage  of  subjects  whose  individual  performance  exceeded  d-prime 
discrimination  greater  than  one. 

Figure  1  illustrates  performance  across  participants.  The  line  graph  (left  y-axis)  represents 
the  group’s  mean  d-prime  performance  across  the  five  factor  spacing  conditions.  As  a  group, 
performance  exceeded  d-prime>1  when  factors  were  spaced  0.762mm  (2  factor  gap).  A  one 
way  ANOVA  revealed  a  significant  main  effect  for  factor  spacing  (F(4,60)=4.466,  p<.005).  As 
the  factor  spacing  increased,  performance  improved.  The  line  graph  (right  axis)  illustrates 
the  percentage  of  subjects  in  each  condition  who  individually  exceeded  threshold.  For 
example,  12.5%  of  subjects  scored  above  threshold  in  the  most  closely  spaced  254p 
condition.  So  while  the  group  average  threshold  is  0.762mm,  there  are  individuals  who  have 
better  than  average  discrimination  abilities.  This  may  suggest  that  an  individual’s  actual  limit 
may  be  better  than  0.254mm  spacing. 


Two  Point  Discrimination 


Figure  1.  Two  Point  Discrimination  Across  Participants 
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2.2  Temporal  Gap  Discrimination 

Gap  discrimination  was  performed  using  a  modified  BrainPort  Balance  Device  C200  (BBD- 
C200)  as  described  above.  Internal  pilot  studies  suggested  temporal  gaps  with  the  smallest 
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array  (0.254mm  spacing)  were  difficult  to  discriminate.  Using  a  linear  array  with  10  tactors 
sized  0.667mm  spaced  1 .0mm  apart,  internal  subjects  were  able  to  feel  temporal  gaps  of  50- 
100ms  in  duration.  With  this  array,  conditions  were  generated  that  could  both  exceed 
performance  and  also  be  within  the  range  of  successful  performance.  All  gap  discrimination 
conditions  use  a  continuous  repeating  pulse  scheme  presenting  1  pulse  for  25ps  wide  every 
125ps  (frame)  to  each  tactor  with  a  programmable  “gap”  of  no  stimulation  on  one  of  the 
tactors.  For  the  “gap”  condition,  the  tactor  pulses  for  a  total  of  500ms  (four  frames)  followed 
by  an  “off  gap”  of  33-250ms. 

Sixteen  subjects  (10  men;  6  women)  participated,  ranging  in  age  from  18-39  years  (mean 
24.5yrs).  Experiment  duration  was  approximately  two  hours.  Subjects  were  recruited  from 
the  Madison,  Wl  area.  All  subjects  gave  informed  consent  with  a  study  protocol  approved  by 
the  New  England  IRB.  Some,  but  not  all,  subjects  participated  in  the  2-pt  Discrimination  Test. 
Pulse  amplitude  (in  volts)  was  determined  individually  before  running  the  gap  discrimination 
experiment.  Subjects  were  presented  with  trials  containing  tactors  firing  on  the  array  with 
variable  gaps  in  the  continuous  stimulation.  They  were  instructed  to  manipulate  a  hand  held 
slider  control,  allowing  each  subject  to  explore  and  adjust  the  stimulation  to  a  comfortable 
working  level.  Once  a  comfortable  level  was  reached,  subjects  indicated  their  response, 
using  either  a  thumb  or  finger  to  press  a  button  on  the  hand-held  control.  “Working  level” 
was  defined  as  a  stimulation  level  that  was  strong  and  comfortable  for  at  least  five  seconds. 
Intensity  values  (voltage)  were  repeated  several  times  and  averaged  across  conditions, 
creating  a  mean  “working  level”  as  the  fixed  voltage  intensity  for  the  gap  discrimination  test. 

This  experiment  consisted  of  five  blocks  of  experimental  runs  containing  twenty  Two- 
Alternative  Forced  Choice  (2AFC)  trials  for  a  total  of  100  trials.  Each  trial  contained  two 
stimulating  tactors,  one  on  the  right  and  one  on  the  left,  spaced  four  tactors  apart.  For  every 
trial,  one  tactor  was  continuously  firing  (as  defined  above),  while  the  other  tactor  had  a 
temporal  gap.  The  targeted  gap  stimulus  was  presented  “on”  for  500ms  followed  by  one  of 
five  “off”  gaps:  33ms,  50ms,  75ms,  100ms  and  250ms.  This  target  stimulus  cycled  through 
“on”  and  “off”  continuously  while  the  other  tactor  presented  a  continuous  waveform.  The  two 
stimuli  were  presented  until  the  participant  indicated  which  stimulating  tactor  had  the 
temporal  gap. 

Subjects  were  instructed  to  press  the  left  button  if  they  felt  the  stimulus  had  a  temporal  gap 
on  the  left,  else,  press  the  right  button  if  they  felt  it  on  the  right.  Half  the  correct  responses 
for  the  trials  for  each  were  presented  on  the  right  and  left  respectively.  Again,  the  participant 
was  allowed  to  take  as  much  time  as  needed  and  a  free  moving  tongue  was  allowed.  In  this 
way,  subjects  could  explore  the  stimulus  with  whatever  part  of  the  tongue  they  felt  gave  them 
the  best  information.  The  order  of  the  five  blocks  was  pseudo-randomized  for  each  subject 
and  there  was  no  performance  feedback  given.  Two  point  discrimination  performances 
translated  into  a  percent  correct  score  for  each  temporal  gap  and  subject.  Effective 
discrimination  threshold  was  defined  as  when  the  percent  correct  was  greater  than  75%. 
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Figure  2  illustrates  performance  across  participants.  The  line  graph  (left  y-axis)  represents 
the  group’s  mean  d-prime  performance  across  the  five  temporal  gap  conditions.  As  a  group, 
performance  exceeded  d-prime>1  when  factors  had  a  gap  of  at  least  75ms.  A  one  way 
ANOVA  revealed  a  significant  main  effect  for  factor  spacing  (F(4,60)=6.239,  p<.005).  As  the 
temporal  gap  increased,  performance  improved.  The  dotted  line  illustrates  the  percentage  of 
subjects  in  each  condition  who  individually  exceeded  threshold.  While  the  group  average 
threshold  is  75ms  for  gap  detection,  there  are  individuals  who  have  better  than  average 
discrimination  abilities.  This  may  suggest  that  an  individual’s  actual  limit  may  be  better  than 
50ms. 


Flicker  Fusion 


n=16  p<0.0001 


Gap  Duration  (ms) 

Figure  2.  Gap  Detection  Across  Participants 


2.3  Tongue  Mapping  &  Spatial  Summation 

A  review  and  analysis  of  the  tongue  mapping  and  spatial  summation  data  using  the  25x25 
tongue  array  suggests  tongue  sensitivity  is  a  function  of  stimulation  location  as  well  as 
number  of  stimulating  factors.  A  representative  three-dimensional  tongue  sensitivity  map  is 
included  ( Figure  3).  The  front  center  of  the  tongue  has  the  greatest  sensitivity,  as  indicated 
by  the  lowest  voltage  region.  As  stimulation  is  placed  near  the  back  of  the  tongue,  the  mean 
sensitivity  is  decreased.  When  many  factors  are  stimulated,  less  voltage  is  required  for 
threshold  and  working  levels.  When  one  factor  is  stimulated,  more  voltage  is  required  for 
threshold  and  working  levels.  The  spatial  summation  graph,  Figure  4,  illustrates  that 
sensitivity  is  a  function  of  the  size  of  the  stimulating  area  on  the  tongue.  In  addition,  the 
graph  illustrates  the  dynamic  range  in  voltage  as  a  function  of  the  number  of  stimulating 
factors,  where  there  is  a  greater  range  when  one  factor  is  firing  as  compared  to  a  smaller 
range  when  nine  factors  are  firing. 
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Figure  3.  Tongue  Sensitivity  Maps 
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Figure  4.  Spatial  Summation 
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2.4  Contrast  Sensitivity 

Using  the  20x20  portion  of  the  High  Density  TUNS  array  (76pm  diameter  tactor,  152pm  center-to- 
center  spacing),  contrast  sensitivity  was  measured  in  five  participants.  Twenty-five  experimental 
blocks  included  five  trials  each  of  five  spatial  line  gratings  presented  at  five  different  contrast  levels: 


5-gap  (760pm) 
4-gap  (608pm) 
3-gap  (456pm) 
2-gap  (304pm) 
1-gap  (152pm) 


100%  Working  Level  Voltage 
80%  Working  Level  Voltage 
60%  Working  Level  Voltage 
40%  Working  Level  Voltage 
20%  Working  Level  Voltage 


Figure  5  illustrates  the  general  experimental  design  (not  to  scale).  Each  experimental  block  consists 
of  15  randomly  presented  grating  orientations:  5  horizontal,  5  vertical  or  5  neither  (diagonal). 
Participants  were  required  to  make  a  Three-Alternative  Forced  Choice  indicating  the  direction  of  the 
presented  grating.  Perceptual  threshold  is  reached  in  conditions  where  percent  correct  exceeds 
66.67%. 
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Figure  5.  Contrast  Sensitivity  Experiment  Design 


A  two-way  ANOVA  (gap-spacing  x  contrast)  was  performed,  resulting  in  a  significant  main  effect  of 
gap  spacing,  Figure  6,  (F(4,16)=6.044,  p<.004),  a  significant  main  effect  of  contrast  level,  Figure  7, 
(F(4,16)=5.406,  p<.006),  with  no  significant  interaction. 
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Figure  6.  Main  Effect  -  Gap  Spacing 
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Figure  7.  Main  Effect  -  Contrast  Voltage 
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Group  percent  correct  values  for  each  experimental  condition  are  shown  in  the  following 
table  (Table  1 ).  Conditions  that  meet  exceed  threshold  are  highlighted  in  yellow. 

Table  1.  Percent  Correct  for  Contrast  Experimental  Conditions 


20% 

760pm 

74.67 

608pm 

57.33 

456pm 

54.67 

304pm 

37.33 

152pm 

40 

40% 

84 

74.67 

57.33 

37.33 

53.33 

60% 

84 

64 

68 

38.67 

58.67 

80% 

82.67 

72 

66.67 

40 

44 

100% 

65.33 

68 

69.33 

42.67 

36.67 

Using  this  data,  a  graphical  “Contrast  Sensitivity”  chart  is  included,  Figure  8,  (not  to  scale), 
with  conditions  that  meet  or  exceed  threshold  are  highlighted  in  yellow.  Two  conditions, 
colored  in  light  green  (100%/760um,  60%,608um),  are  close  to  threshold  and  one  would 
assume  with  more  subjects  would  be  included  as  above  threshold. 
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Figure  8.  Contrast  Sensitivity  Chart 
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3  HARDWARE  AND  SOFTWARE  DEVELOPMENT 


Hardware  development  over  the  course  of  this  project  went  from  simple  linear  arrays  for 
initial  spatial  discrimination  testing  to  a  final  20x100  High  Density  Array  and  simultaneous 
control  system.  Recent  engineering  effort  for  the  this  program  has  been  focused  on  building 
the  High  Density  (HD)  Array  Control  System,  which  is  now  in  use  for  discrimination  testing. 

The  overall  System  Architecture  is  shown  below,  Figure  9.  This  system  provides  micro¬ 
electrode  transcutaneous  neuro-stimulation  on  a  new  scale:  thousands  of  simultaneous 
active  electrodes  and  up  to  96  unique  waveforms  (limited  only  by  memory). 

The  Experiment  Workstation  runs  the  experiment  control  software  application  and  provides  a 
Common  Software  Interface,  an  Ethernet  Connection  to  Control  Modules,  a  Socket 
communications  protocol.  It  is  designed  to  scale  with  the  control  hardware. 

The  HD  Controller  Boards  are  designed  to  manage  100  electrodes  each.  They  include  a 
Linux-based  Microcontroller  (Gumstix)  executing  custom  control  software  (the  Experiment 
Virtual  Machine),  an  FGPA  (10MHz  waveform  clock)  providing  Master-Slave  synchronization 
across  all  control  boards,  simultaneous  electrode  activation,  and  the  electrode  drivers.  The 
control  boards  are  designed  to  allow  each  electrode  to  be  an  active  electrode  (stimulating),  a 
return  electrode,  or  floating  (out  of  the  circuit). 
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Figure  9.  High  Density  Array  Control  System  Architecture 
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Figure  10  is  a  block  diagram  of  the  core  hardware  architecture. 
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Figure  10.  Core  Hardware  Architecture 
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3.1  Hardware 

Significant  Accomplishments:  Design  and  manufacture  of  the  HD  Control  boards.  Each 
board  manages  the  state  of  100  electrodes  and  allows  simultaneous  activation  of  any  or  all  at 
a  given  time.  Figure  1 1  shows  a  completed  HD  Control  board.  We  currently  have  22  boards 
in  house. 
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Figure  11.  Complete  HD  Control  Board 


■  To  control  the  array  stack-up  described  below,  an 
HD  Control  board  is  required  for  every  100 
electrodes.  Figure  12  shows  a  board  being  inserted 
into  the  instrumentation  rack.  The  initial  system  will 
handle  2000  electrodes  (20  control  boards).  Each 
board  has  a  cable  connecting  it  to  a  set  of  electrodes 
(cables  are  seen  hanging  off  the  front  of  several 
boards). 

Figure  12.  Control  Board  Insertion 
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■  When  fully  populated,  the  instrumentation  rack,  Figure  13,  contains  the  electro¬ 
mechanical  interfaces  allowing  the  Experiment  Workstation  (a  PC  with  custom 
software)  to  control  each  HD  board  and  every  tactor. 


Figure  13.  Populated  Instrumentation  Rack 


This  initial  implementation  was  sufficient  to  support  the  planned  experiments.  During  the 
course  of  the  project,  Wicab  and  DARPA  agreed  to  limit  testing  to  devices  with  2000 
electrodes  before  deciding  whether  fabrication  of  larger  capacity  arrays  is  necessary  (Note 
that  the  system  is  designed  to  scale  up  to  20,000+  electrodes). 

Wicab  used  this  high  density  array  system  to  conduct  studies  at  the  limits  of  spatial,  temporal 
and  contrast  discrimination  (Figure  14).  In  addition,  the  control  architecture  supports 
waveform  shaping  and  simultaneous  activation  of  many  electrodes  (compared  to  previous 
systems  where  only  one  electrode  at  a  time  is  activated). 
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High  Density  Array  Stack-up 

Figure  15. (a)  shows  the  completed  proof  of  concept  prototype  validating  the  design  and 
fabrication  approach  to  the  assembly. 


(a)  (b) 


Figure  15.  Strip  Array  Stack-up 

At  arms  length  the  individual  electrodes  are  barely  visible  (Figure  15.b)  to  the  naked  eye. 
Figure  16  shows  a  single  strip  array,  with  a  20  layer  stack  of  strips. 
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Figure  16.  Stack-up  and  Single  Strip  Array 


3.2  Software 

Software  modules  have  been  developed  for  the  experiment  system  throughout  the  TUNS 
project  to  support  psychophysical  experiments  and  provide  the  necessary  mechanisms  to  run 
subjects  on  two  tongue  stimulation  platforms  (WG-C200  and  HD  Controller).  Data  formats 
were  also  developed  to  encode  experiment  parameters,  waveform  parameters/timings, 
stimulus  patterns,  etc.  for  use  with  the  custom  software  being  developed.  The  data  formats 
and  software  modules  designed  during  this  project  are  summarized  in  this  document. 
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3.3  Stimulation  Platform 


3.3.1  Virtual  Machine  Concept 

In  order  to  describe  stimulation  parameters  in  a  generic  enough  format  to  produce  similar 
system  behavior  across  hardware  implementations,  a  virtual  machine  design  has  been  used. 
Tongue  stimulation  patterns/timings  are  parameterized  using  a  custom  instruction  set  that 
executes  in  the  virtual  machine  via  a  scheduled  run-to-completion  scheme.  Programs  written 
using  the  instruction  set  are  described  as  „Stimulus  Programs’  (Figure  17).  A  fixed  master 
clock  drives  the  virtual  machine,  synchronizing  voltage  outputs  at  the  tactors.  Only  a  few  of 
the  instructions  are  hardware  dependent  (those  related  to  actually  commanding  the  hardware 
voltage  outputs  and  waveform  selections)  implying  that  the  virtual  machine  can  easily  be 
ported  to  various  hardware  platforms. 
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Figure  17.  Stimulus  Program  WorkFlow 


3. 3. 1.1  Instruction  Set 

The  instruction  set  is  outlined  in  detail  in  Appendix  A  -  Virtual  Machine  Instruction  Set. 
Instructions  have  been  designed  for  such  tasks  as  tactor  addressing,  voltage  adjustment, 
waveform  selection,  clock  adjustment  and  simple  execution  control  (looping,  conditionals, 
etc.).  When  used  to  create  a  Stimulus  Program,  execution  of  the  instructions  provides  fully 
programmable  voltage  pulses  at  the  tactor  array/IOD  limited  only  by  the  underlying  hardware 
implementation. 

3.3. 1.2  VM  State 

The  virtual  machine  state  is  outlined  in  detail  in  Appendix  B  -  Virtual  Machine  State 
Definition.  In  order  for  the  virtual  machine  to  execute  the  instructions  in  a  Stimulus  Program, 
an  associated  virtual  machine  state  is  maintained.  The  state  variables  are  referred  to  as 
..registers’  since  they  serve  the  same  purpose  as  processor  registers  for  the  virtual  machine. 
These  registers  are  broken  down  into  several  subsets;  timing  registers,  loop  control  registers, 
address  registers,  voltage  registers,  count  registers,  block  voltage  registers,  loop  control 


19 


registers,  waveform  registers  and  hardware  specific  configuration  registers.  Stimulus 
Programs  can  use  these  registers  to  affect  how  the  voltage  at  a  tactor  is  changed  over  time. 

3. 3. 1.3  Stimulus  Programs 

A  Stimulus  Program  defines  a  list  of  instructions  designed  to  produce  a  specific  stimulus 
timing  pattern  at  the  tactor  outputs.  Stimulus  Programs  define  exactly  how  the  tactor  outputs 
will  be  adjusted  over  time  based  on  values  stored  in  the  VM  State  registers  and  the  master 
clock  rate.  The  stimulus  timing  and  voltage  output  features  are  limited  only  by  the  underlying 
hardware  capabilities  such  as  maximum  clock  speed  and  stimulation  power  supply. 

3.3.1 .4  Hardware  Specific  Features 

For  each  platform  for  which  the  virtual  machine  is  implemented,  a  small  set  of  instructions 
must  be  implemented  uniquely  based  on  the  underlying  stimulation  hardware.  For  example, 
a  WG-C200  device  uses  a  single  DAC  channel  to  adjust  voltage  output  and  a  Multiplexer  to 
select  the  single  tactor  that  will  receive  the  voltage  output.  In  order  for  the  Stimulus  Program 
to  adjust  voltage  on  the  platform,  custom  code  must  be  implemented  to  communicate  with 
the  DAC  and  Multiplexer.  The  HD  Controller  voltage  output  cannot  be  adjusted  directly, 
however  it  has  the  added  feature  of  custom  waveform. 

3.3.2  HD  Controller 

In  order  to  support  high  density  arrays,  a  controller  was  designed  to  allow  parallel  boards  to 
drive  a  set  of  simultaneously-firing  factors,  synchronized  by  a  master  clock.  The 
implementation  developed  for  this  project  utilizes  a  Gumstix  ultra-mobile  single-board- 
computer  attached  to  a  custom-fab  board  containing  an  FPGA,  SRAM  and  associated  drive 
electronics  to  drive  100  tactor  outputs  simultaneously  at  up  to  10MHz. 

The  boards  communicate  with  the  workstation,  acting  as  server,  via  100Mbit  Ethernet. 
Software  on  each  Gumstix  board  executes  a  client  application.  To  drive  factors,  each  board 
has  a  100  pin  interconnect  onto  which  a  ribbon  cable  can  be  attached.  On  the  other  side  of 
the  ribbon  cable,  a  number  of  different  array  configurations  can  be  connected  depending  on 
the  requirements  of  the  experiment.  For  small  linear  arrays,  a  subset  of  the  boards  may  be 
used  to  drive  a  small  number  of  factors  (e.g.  100  factors  can  be  driven  with  a  single  board). 
For  higher  density  arrays,  any  number  of  boards  up  to  a  total  of  20  on  the  currently  built 
system  can  be  utilized  to  drive  up  to  2000  factors. 

All  boards  are  linked  by  a  master  clock  bus  and  one  board  is  specified  as  the  master  to  drive 
the  master  clock.  The  FPGA  on  the  master  board  outputs  a  programmable  PWM  signal  to  all 
other  FPGAs  on  slave  boards  to  synchronize  tactor  outputs  across  all  boards.  Each  board 
has  an  SRAM  directly  attached  to  the  FPGA  to  store  waveform  data.  Waveform  data 
consists  of  X/Y/Z  states  (X/Y/Z  states  correspond  to  high/low/ground)  for  the  transistor  on  a 
given  tactor  over  time  producing  a  “1-bit  DAC”  style  output  on  the  tactor  when  stimulation 
voltage  is  fixed.  The  internal  clock  on  the  FPGA  provides  transistor  state  switching  at  up  to 
10MHz  when  clocking  waveform  data  from  the  on-board  SRAM. 

The  virtual  machine  implementation  on  the  Gumstix  boards  provides  methods  (via  WGIO 
interface  over  Ethernet)  to  program  the  FPGA,  load  waveform  data  into  the  SRAM,  set  the 
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master  clock  frequency,  assign  waveform  selections  per  tactor  and  control  the  stimulation 
power  relay  (on/off).  The  software  on  the  Gumstix  also  provides  feedback  to  the  workstation 
via  a  simple  web-based  interface  (HTTP).  The  web  interface  provides  feedback  as  to  the 
status  of  the  virtual  machine  and  has  been  used  for  debugging  and  verification  that  the 
intended  waveforms  are  being  sent  to  the  tactors. 

3.3.3  WG-C200 

To  support  early  experiment  execution  using  available  hardware,  custom  software/firmware 
was  developed  for  the  BrainPort  Balance  Device  C200.  A  device  executing  the  custom 
software/firmware  is  referred  to  as  WG-C200.  The  WG-C200  device  is  capable  of  stimulating 
a  single  tactor  at  a  time  when  used  with  discrete  lODs  (one  wire  per  tactor)  or  up  to  4  tactors 
at  a  time  when  used  with  row/column  based  arrays.  Voltage  output  level  is  adjustable  by  a 
DAC  and  the  active  tactor  is  selected  using  a  multiplexer  (4  DAC  channels  are  used  on 
row/column  arrays). 

The  WG-C200  device  communicates  using  the  WGIO  protocol  (see  Appendix  C  -  WGIO 
Packet  Definitions)  with  the  workstation  via  an  RS232  serial  cable  @  115200  baud.  Up  to 
100  tactors  can  be  driven  when  connected  to  the  120  pin  connector  on  the  device  (for 
discrete  arrays).  Row/column  arrays  can  be  driven  with  a  row/column  adapter  and  an 
appropriately  programmed  device  (supports  both  18x18  and  25x25  row/column  based 
arrays).  For  small  linear  arrays,  a  subset  of  the  100  tactor  outputs  may  be  used  to  drive  a 
small  number  of  tactors  (e.g.  1 0  tactor  linear  arrays).  The  DAC  can  be  adjusted  in  timing 
increments  down  to  lOpsec  to  produce  a 

•  DAC  and  Waveform  Timing 

•  Virtual  Machine  Implementation 


3.4  Software  Modules 

The  software  modules  developed  for  the  experiment  system  can  be  broken  into  two 
subsections,  pc-based  (workstation)  and  embedded  (HD  Controller).  The  windows  software 
applications  provide  an  interface  to  the  researcher  and/or  developer  for  executing 
experiments  (Experiment  Controller)  and  testing  system  operation  (HD  Diagnostic  Utility). 


3.5  Workstation  PC  Software  Modules 

3.5.1  Experiment  Controller 

The  Experiment  Controller  application  reads  configuration  from  a  Trial  Configuration  file  (see 
the  Data  Formats  section  for  more  details).  The  researcher  selects  an  experiment  (with  a 
corresponding  Trial  Configuration  file)  and  steps  through  the  software.  Experiment  specific 
parameters  are  entered  by  the  researcher  and  the  subject  is  presented  with  a  set  of  trials  for 
which  they  respond  using  an  input  device.  The  Experiment  Controller  software  records  the 
subject  response  for  each  trial  and  generates  an  experiment  report  upon  completion.  In 
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order  to  present  trials  to  the  subject,  the  Experiment  Controller  communicates  with  the  HD 
Controller  sending  it  trial-specific  parameters  which  determine  the  stimulus  pattern  and 
waveforms  used,  Figure  18.  Experiment  Controller  also  controls  logic  and  stimulation  power 
supplies  for  the  HD  Controller  via  RS232  serial  link. 


Experiment  Controller  HDServer/Client  communication 
steps  during  execution  of  a  trial 


Figure  18.  HD  Server/Client  Communication 


3.5.2  Utilizing  the  VM  to  Present  Trials 

The  „Virtual  Machine’  running  on  the  HD  Controller  system  (see  the  Virtual  Machine  section 
for  more  details)  controls  communications  with  the  on-board  FPGA/SRAM  and  sequencing  of 
factor  selections  to  the  attached  IOD.  This  is  accomplished  through  execution  of  a  Stimulus 
Program  in  a  scheduled  run-to-completion  scheme.  Stimulus  patterns  and  timings  are 
defined  as  parameters  to  the  Stimulus  Program  designed  for  a  particular  experiment.  For 
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each  trial  in  an  experiment,  the  Experiment  Controller  sends  down  appropriate  stimulus 
parameters  to  cause  the  Stimulus  Program  to  present  the  intended  stimulus  to  the  IOD. 


3.5.2. 1  Subject  Input  Devices 

Two  forms  of  subject  input  device  are  possible  with  the  Experiment  Controller;  the  hand-held 
controller  and  the  numeric  keypad.  The  Trial  Configuration  file  for  a  given  experiment 
defines  which  subject  input  device  is  to  be  used. 

3. 5. 2. 2  Power  Supply  Control 

The  HD  system  includes  two  programmable  power  supplies;  the  logic  power  supply  (powers 
logic  circuits  on  the  HD  Controller  boards)  and  the  stimulation  power  supply  (powers  the 
waveforms  to  the  IOD). 

3. 5. 2. 3  Platform  Specific  Operations 

The  Experiment  Controller  contains  several  features  specific  to  the  WG-C200  platform  such 
as  serial  port  communications  and  use  of  „DAC’  values  as  intensity  control.  It  also  includes 
features  specific  to  the  HD  Controller  system  such  as  power  supply  control  and  ramping 
features,  support  for  waveform  data  transfer  and  FPGA  file  support. 

3.5.3  HD  Diagnostic  Utility 

The  HD  Diagnostic  Utility  began  as  a  utility  to  visualize  Trial  Profiles  (see  Trial  Configuration 
section  for  more  details)  to  ensure  accuracy  when  developing  tactor  selection  patterns  on 
high  density  arrays  (up  to  2000  tactors).  This  utility  provides  a  method  to  select  waveforms 
per  tactor  in  a  similar  fashion  to  a  paint  program  and  allows  importing  of  images  to  create  a 
Trial  Profile.  Other  features  include  communication  with  HD  Controller  boards  and  functions 
to  test  operations  such  as  sending  waveform  data,  FPGA  data  or  Stimulus  Program  data  to 
the  board(s). 

3.5.4  HD  Controller  Library  and  the  WGIO  Protocol 

The  HD  Controller  Library  implements  the  windows  side  of  the  WGIO  Communications 
Protocol.  This  protocol  provides  a  set  of  packet  definitions  allowing  Windows  applications  to 
command  the  HD  Controller  to  start/stop  stimulation,  update  the  Waveform  Definitions, 
upload  an  FPGA  Program  file,  update  Stimulus  Programs  and  assign  many  other  parameters 
affecting  how  stimulation  is  presented  to  the  IOD.  The  WGIO  protocol  provides  a  handle  into 
the  state  machine  of  the  Virtual  Machine  executing  on  the  HD  Controller  boards.  For  a 
detailed  list  of  WGIO  packets,  see  Appendix  B  -  WGIO  Protocol  Packet  Definitions.  The  HD 
Controller  Library  provides  an  event-based  I/O  model  to  the  parent  application  and 
implements  a  socket-based  communications  layer  to  the  HD  Controller  boards  via  Ethernet 
acting  as  a  server  in  a  client-server  model. 
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3.6  HD  Controller  Board  Modules 


3.6.1  Gumstix  Platform 

Each  HD  Controller  board  requires  a  microprocessor  to  configure  the  FPGA  to  sequence  the 
tactor  outputs  and  communicate  with  the  PC  Workstation.  For  this  task,  a  „Gumstix’  brand 
single-board  computer  has  been  chosen  for  its  speed,  size  and  versatility  in  communications 
methods  (Ethernet,  serial,  address/data  bus,  etc.).  The  board  includes  a  32-bit  Marvell 
PXA270  processor  running  at  600  MHz  with  1 28MB  SDRAM  and  64MB  flash.  The  board 
runs  embedded  Linux  providing  TCP/IP  communication  and  Compact  Flash  support  for  ease 
of  development  and  integration.  The  FPGA  is  connected  to  the  PXA270  via  its  Address/Data 
bus  as  well  as  a  JTAG  interface  from  which  the  Gumstix  board  programs  the  FPGA  during 
the  boot  process. 

3. 6. 1.1  HD  Client  Application 

The  primary  software  application  running  on  the  Gumstix  platform  is  the  HD  Client 
application.  This  software  implements  the  virtual  machine,  programs  and  configures  the 
FPGA  based  on  remote  TCP/IP  communications  via  the  WGIO  protocol  and  provides  an 
HTML  based  status  web  page.  This  software  maintains  communications  (as  client)  with  the 
Experiment  Controller  (as  server)  on  the  PC.  As  an  experiment  is  run,  the  Experiment 
Controller  sends  commands  via  the  WGIO  protocol  to  the  HD  Client  application  and  the  state 
of  the  FPGA  is  updated  to  reflect  the  commanded  changes.  These  changes  to  the  state  of 
the  FPGA  are  moderated  by  the  virtual  machine  as  it  ticks  through  scheduled  run-to- 
completion  steps  of  a  selected  Stimulus  Program. 

3. 6. 1.2  Client  Diagnostic  Application 

The  Client  Diagnostic  application  was  developed  in  the  early  stages  of  testing  the  FPGA  to 
provide  an  easy  interface  to  changing  memory  locations  before  the  full  „HD  Client’  application 
was  developed.  Features  include  control  over  PWM  output  of  the  FPGA,  waveform  selection 
per  tactor,  loading  waveform  data  from  a  file  to  the  SRAM  attached  to  the  FPGA  and  LED 
output  control.  This  utility  is  mainly  useful  for  debugging/troubleshoot  HD  Controller  boards 
exhibiting  unexpected  behavior. 

3. 6. 1.3  Virtual  Machine  Implementation 

The  implementation  of  the  virtual  machine  (VM)  on  the  Gumstix  platform  provides 
implementations  of  all  “general  purpose”  and  several  hardware  dependent  instructions.  Each 
HD  Controller  board  provides  a  trigger  input  which  triggers  execution  of  the  virtual  machine 
“ticks”.  The  primary  instructions  that  support  tactor  output  on  the  HD  system  are  related  to 
waveform  selections  for  each  tactor  via  the  Block  Voltage  registers,  Master  Clock/PWM 
frequency  control  via  the  Tick  Unit  register  and  control  of  starting/stopping  stimulus  programs 
to  sequence  waveform  selections. 

3. 6. 1.4  Ethernet  Communications 

The  set  of  HD  Controller  boards  each  has  a  standard  Ethernet  jack  on  it  attached  to  the 
Gumstix  board  controlling  it.  These  Ethernet  jacks  are  connected  to  a  24  port  switch  to  form 
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a  local  network  along  with  the  workstation  PC  and  the  hand-held  controller.  The  Experiment 
Controller  (as  server)  communicates  with  each  board  individually  (as  client)  via  TCP/IP.  The 
Experiment  Controller  (as  client)  communicates  with  the  hand-held  controller  via  TCP/IP. 

3. 6. 1.5  DHCP  server 

The  PC  Workstation  runs  a  simple  DHCP  server  that  supports  fixing  IP  address  to  MAC 
address  for  each  HD  controller  board.  The  subnet  used  for  the  local  communications  is 
172.16.5.0/255.255.255.0.  The  hand-held  controller  address  is  fixed  at  172.16.5.1  and  the 
PC  workstation  is  fixed  at  172.16.5.5.  The  HD  Controller  boards  are  assigned  IP  addresses 
sequentially  board  0  through  19  respectively  numbered  172.16.5.10  through  172.16.5.29. 

3. 6. 1.6  PC  to  HD  Client  Connections 

Upon  booting  up  and  running  the  “HD  Client”  application,  each  HD  Controller  board  attempts 
a  TCP/IP  connection  (as  client)  to  the  Experiment  Controller  (as  server)  on  the  PC 
workstation  on  TCP.  Upon  connecting,  the  client  communicates  via  the  WGIO  protocol  and 
waits  for  commands  from  the  Experiment  controller. 

3. 6. 1.7  HHC  to  PC  Connection 

Upon  booting,  the  hand-held  controller  listens  (as  server)  for  connections  from  the 
Experiment  Controller  (as  client).  Once  connected,  the  hand-held  controller  sends  ASCII 
formatted  text  packets  reporting  the  status  of  the  slider,  knobs  and  buttons  as  states  change. 
The  Experiment  Controller  can  use  this  data  to  record  a  subject’s  response  to  a  stimulus. 

3. 6. 1.8  Linux  Kernel  Module  and  Interrupts 

To  maintain  synchronized  operations  across  boards,  an  interrupt  is  triggered  with  each  tick  of 
the  Master  Clock/PWM  line. 

3. 6. 1.9  WGTrigger  Kernel  Module 

The  Master  Clock/PWM  line  is  used  to  synchronize  operations  across  HD  Controller  boards. 
This  line  triggers  both  the  FPGA  to  start  a  waveform  pulse  and  the  PXA270  on  the  Gumstix 
board  via  a  GPIO  pin  to  tick  the  VM.  A  Linux  kernel  module  (wgtrigger.ko)  has  been 
developed  to  monitor  GPIO  22  to  which  the  Master  Clock  line  is  connected  and  trigger  an 
interrupt  on  the  rising  edge.  This  interrupt  triggers  the  kernel  module  to  update  waveform 
selections  in  the  FPGA  via  writes  to  the  Address/Data  bus  and  signal  the  virtual  machine  to 
execute  its  next  “tick”  in  its  currently  selected  Stimulus  Program. 

3.6.1.10  /proc  Filesystem 

The  wgtrigger.ko  kernel  module  utilizes  a  custom  file  /proc/wgtrigger  to  provide  a  pipe  for 
communication  between  user  and  kernel  space.  The  kernel  module  uses  it  to  signal  the  HD 
Client  application  to  execute  the  next  VM  tick.  The  HD  Client  application  uses  it  to  update 
the  set  of  waveform  selections  for  each  factor  to  be  presented  on  the  next  Master 
Clock/PWM  tick. 


25 


3.6.2  Hardware  Interconnects 


3.6.2. 1  Address/Data  Bus 

The  FPGA  and  PXA270  are  connected  via  a  26  bit  address  and  32  bit  data  bus.  A  16  bit 
Address/6x16  bit  data  bus  attaches  the  6  Static  RAM  (SRAM)  chips  to  the  FPGA  (See  TR- 
070054  for  more  details).  Address  locations  have  been  defined  within  the  FPGA  for  various 
features  corresponding  to  logic  blocks  that  have  been  programmed  into  it  for  factor  control, 
LED  output,  Master  Clock/PWM  generation  and  waveform  data  storage  to  the  SRAM. 

3. 6. 2. 2  JTAG  Interface 

A  JTAG  interface  (TDO/TDI/TCK/TMS)  is  used  to  program  the  FPGA.  In  order  for  the 
PXA270  to  program  the  FPGA,  4  GPIO  pins  have  been  tied  to  the  FPGA  programming  pins 
and  a  customized  version  of  Lattice’s  ispvm_ui  programming  tool  has  been  developed  for  the 
Gumstix  platform. 

3. 6. 2. 3  Tactor  Interface 

Each  HD  Controller  board  has  a  100  pin  connector  that  interfaces  to  a  100  conductor  ribbon 
cable.  On  the  opposite  side  of  the  ribbon  cables  is  an  adapter  providing  connection  to  either 
a  10  tactor  linear  array  (similar  to  that  used  in  the  WG-C200  device)  or  a  high-density  flex 
array. 

3.6.3  Board-to-Board  Waveform  Synchronization 

Each  HD  Controller  board  supports  a  total  of  100  factors.  In  order  to  support  greater  than 
100  factors  in  parallel,  a  set  of  boards  uses  a  Master  Clock/PWM  signal  to  synchronize  its 
tactor  outputs. 

3.6.4  Master  Clock/PWM  Signal  Output 

One  HD  Controller  within  the  set  is  configured  as  “Master”  which  drives  its  Master 
Clock/PWM  signal  as  an  output  (all  other  boards  are  configured  as  “Slave”  which  disconnects 
their  Master  Clock  Signal  driver).  The  signal  frequency  and  duty  cycle  is  programmable, 
typically  50%  duty  cycle  running  at  <=  1  kHz. 

3.6.4. 1  Responding  to  PWM  Signal  Input 

The  FPGA  and  PXA270  are  both  configured  to  respond  to  opposite  edges  of  the  Master 
Clock/PWM  signal.  The  FPGA  starts  firing  tactor  output  waveforms  on  a  falling  edge  of  this 
signal.  The  PXA270  triggers  a  “tick”  execution  in  its  VM  and  loads  up  waveform  selections  to 
the  FPGA  in  preparation  for  the  next  falling  edge  where  the  output  will  be  presented. 

3.6.5  Waveform  SRAM 

Waveforms  are  generated  on  tactor  output  by  sequencing  the  “high”,  “low”  and  “ground” 
transistors  within  the  signal  path.  These  3  values  are  controlled  as  configuration  bits  that  can 
be  changed  at  a  rate  up  to  10MHz  by  the  FPGA.  To  toggle  these  lines  over  time,  the  FPGA 
sequences  through  a  series  of  addresses  on  the  attached  SRAM  modules  and  clocks  out  the 
data  stored  in  the  SRAM  directly  to  the  transistors  for  each  tactor. 
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3.6.5. 1  Writing  to  SRAM  from  the  Gumstix 

The  SRAM  consists  of  6  individual  chips  with  1 6  bits  of  address/1 6  bits  of  data  64k  deep. 

The  FPGA  provides  an  address  window  to  the  Gumstix  for  writing/reading  to/from  the  SRAM. 
From  the  Gumstix  perspective,  the  SRAM  looks  like  a  contiguous  chunk  of  memory  as  seen 
below,  Figure  19. 
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Figure  19.  SRAM  Memory  Map 


3. 6. 5. 2  Waveform  Selection  Multiplexer 

A  section  of  address  space  1 00  bytes  in  size  @  0x1 01 00000  has  been  defined  on  the  FPGA  to 
specify  which  of  the  128  waveforms  in  the  waveform  table  should  be  output  on  each  tactor.  The 
Gumstix  writes  25  values  (32  bits  each)  to  change  the  waveform  that  is  presented  to  each  tactor. 

3. 6. 5. 3  Clocking  of  Waveforms  to  Transistors 

SRAM  data  starting  at  address  0x10000000  is  clocked  out  to  all  transistor  inputs  (X/Y/Z)  in  a  series  of 
4  clock  cycles  on  the  FPGA’s  40MHz  clock.  On  a  falling  edge  of  the  Master  Clock/PWM  signal,  the 
FPGA  starts  its  waveform  output  sequence  sequentially  clocking  out  data  from  SRAM  to  the 
transistors  for  800ps.  The  data  in  the  SRAM  is  configured  by  the  Gumstix  via  the  address/data  bus 
before  the  Master  clock/PWM  signal  is  started.  By  default,  the  first  waveform  in  any  waveform  table 
should  be  programmed  to  select  the  “ground”  value  for  all  ticks  within  the  waveform  in  order  to  ensure 
that  the  system  can  always  disable  stimulation  by  selecting  waveform  0  for  all  tactors. 

3. 6. 5.3. 1  Master  Waveform  Clock 

The  Master  Waveform  clock  runs  at  1 0Mhz  derived  from  the  40MHz  clock  rate  of  the  FPGA.  It  takes 
4  clock  cycles  to  clock  the  X,  Y  and  Z  states  out  to  all  100  tactors  (100  x  3  bit  channels).  This  is 
derived  from  1  bit  for  X,  1  bit  for  Y,  1  bit  for  Z  needing  to  be  clocked  out  for  100  different  tactors  (3  x 
100  bit  channels  written  out  128  bits  at  a  time  in  4  clock  cycles). 


3. 6. 5. 3. 2  Waveform  Data  Length  Limitations 

The  depth  of  the  SRAM  is  64K  which  determines  the  maximum  length  of  a  waveform  that  can  be 
presented  to  the  tactors.  The  configuration  chosen  provides  storage  for  up  to  128  unique  waveforms 
up  to  1.6ms  in  length.  This  is  derived  from  6  chips  x  16  bits  x  64K  depth  =  6291456  bits  of  data 
(786432  bytes)  divided  by  128  waveforms  divided  by  3  bits  per  tick  giving  16384  ticks  worth  of 
transistor  states  stored.  Since  the  waveform  clock  is  effectively  10MHz,  16384  ticks  will  take 
1 ,6384ms. 
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3.6.6  Power  Supply  Control 

To  power  the  HD  Controller  system,  two  separate  power  supplies  are  used.  Each  supply  is 
plugged  into  a  hospital  grade  isolation  transformer  to  isolate  the  subject.  Each  power  supply 
is  connected  to  the  PC  workstation  via  RS232  link.  This  RS232  connection  allows  the 
Experiment  Controller  and  other  utility  software  to  control  the  current  limits  and  voltage  level 
for  the  supplies  remotely. 

3.6.6. 1  Logic  Power  Supply 

The  logic  power  supply  provides  a  constant  5.0V  to  the  HD  Controller  boards  to  power  the 
Gumstix  boards,  FPGA  and  related  hardware  via  on-board  regulators. 

3. 6. 6. 2  Stimulation  Power  Supply 

The  stimulation  power  supply  correlates  to  the  stimulation  level  that  is  output  at  the  IOD.  The 
supply  can  be  adjusted  during  an  experiment  to  find  a  level  that  is  perceived  as  “comfortable 
working  level”  to  a  subject. 

3.6.7  DIP  Switches/LED  Indicators 

3.6.7. 1  FPGA  Programmed  Indicator 

The  blue  LED  turns  on  after  the  FPGA  has  been  successfully  programmed. 

3. 6. 7. 2  Stimulation  Active  Indicator 

The  amber  LED  turns  on  anytime  the  stimulation  relay  is  open  and  providing  stimulation. 

3. 6. 7. 3  Stimulation  Supply  Charged  Indicator 

The  yellow  LED  on  the  front  of  the  HD  Controller  board  indicates  when  the  capacitors  are 
charged  and  ready  to  stimulate.  This  LED  fades  as  the  capacitors  drain. 

3. 6. 7. 4  Master/Slave  Indicator 

The  red  LED  indicates  whether  an  HD  Controller  board  is  configured  as  a  master  or  slave.  If 
a  board  is  configured  as  master,  the  red  LED  will  be  on  and  its  Master  Clock/PWM  output  will 
be  connected,  otherwise  the  red  LED  will  be  off. 

3. 6. 7. 5  Master/Slave  Switch 

The  first  DIP  switch  is  used  to  set  whether  the  HD  Controller  board  will  be  configured  as 
master  or  slave.  If  the  board  is  used  on  its  own,  it  must  be  configured  as  master  and  a 
jumper  must  be  installed  in  the  jumper  block  to  complete  the  Master  Clock/PWM  loop-back 
connection. 

3.6.8  FPGA 

3.6.8. 1  PWM  Signal  Programming 

The  Master  Clock/PWM  signal  is  programmable  from  the  Gumstix  at  address  0x10300000. 
The  PWM  signal  is  generated  using  two  counters  which  determine  the  “high  time”  and  the 
“low  time”  for  the  signal  based  on  the  40MHz  base  clock.  This  provides  the  ability  to  program 
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the  PWM  output  with  quite  high  resolution  in  the  required  frequency  range  for  the  system 
(typically  <=  1  kHz). 

3. 6. 8. 2  Waveform  Selection  Multiplexer 

The  FPGA  has  a  set  of  multiplexers  providing  the  ability  to  select  any  one  of  the  1 28 
waveforms  loaded  into  the  SRAM  for  output  to  any  tactor.  This  implies  that  each  tactor  can 
have  a  completely  unique  waveform  or  can  be  programmed  to  have  the  same  waveform  as 
other  tactors  based  on  the  multiplexer  input  values.  To  specify  the  waveform  that  is  used  for 
a  given  tactor,  an  8  bit  value  must  be  written  to  the  corresponding  location  in  memory 
location  0x10100000. 

3. 6. 8. 3  LED  Indicator  Controller 

The  FPGA  provides  access  to  changing  the  state  of  the  output  LEDs  as  bits  in  a  bitfield  at 
location  0x10200000. 


3.7  JTAG  Programming 

•  Lattice  semiconductor  provides  source  code  for  a  JTAG  programming  module 
(ispvm_ui)  that  could  be  customized  to  work  with  the  HD  Controller  platform.  Four 
GPIO  pins  available  on  the  Gumstix  board  were  programmed  as  I/O  and  the  ispvm_ui 
tool’s  source  code  was  configured  to  utilize  these  pins  for  TDI,  TDO,  TCK  and  TMS. 

•  The  HD  Client  application  implements  a  command  that  can  be  triggered  from  the 
Experiment  Controller  to  transfer  an  FPGA  program  in  the  form  of  a  .VME  file.  Once 
the  .VME  file  is  transferred  to  the  HD  Client  application,  the  ispvm_ui  command  line 
utility  is  called  to  perform  the  programming  step  placing  the  .VME  code  into  the  FPGA 
via  the  JTAG  interface. 
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Deprecated  Software  Modules 

As  of  the  development  of  the  HD  Controller  system,  the  WGC200  platform  is  no  longer  used 
to  collect  data.  The  software  modules  developed  for  the  WGC200  are  frozen  and  will  be 
considered  ..deprecated’  as  we  do  not  plan  to  further  develop  the  software.  Software 
modules  developed  for  the  WGC200  platform  include: 


Software  Module 

Description 

WG-C200  Firmware 

Provides  a  subset  of  the  Wave  Generator  I/O  (WGIO) 
interface  enabling  experiments  to  be  developed  for  a 
modified  BrainPort  Balance. 

WG-C200  Simulator 

Simple  waveform  simulator  running  on  Windows  that 
outputs  tactor  stimulation  levels  graphically.  This 
application  was  no  longer  needed  once  the  WG-C200 
firmware  was  verified  to  be  working  properly. 

Array  Controller  API 
Test 

Simple  Windows  Forms  application  providing  a  graphical 
interface  to  the  WGIO  protocol  sending/receiving  packets 
to/from  the  WG-C200  device. 

Trial  Configurator 

This  tool  was  used  to  assist  in  creating  the  two  point 
discrimination  experiment  for  the  WG-C200  but  was  never 
finished. 

3.8  Data  Formats 

o  Trial  Configuration 

■  Stimulus  Program 

■  FPGA  Program  Files 

■  Waveform  Definitions 
o  Experiment  Controller  Report 
o  IOD  Definition  Files 


4  PUBLICATIONS 

Arnoldussen,  A.,  Hogle,  R.,  Fisher,  F.,  Lederer,  S.,  Rosing,  M.,  Besta,  M.,  Ferber,  A.  (2008) 
Spatial  Resolution  on  the  Tongue  as  Applied  to  a  Prosthetic  BrainPort®  Vision  Device.  ARVO 
Abstr.  2898/A101. 
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APPENDIX  A- VIRTUAL  MACHINE  INSTRUCTION  SET 


Instruction 

Arguments 

Description 

AC 

Set  address  to  a  constant  value 

AR 

Set  address  to  a  register  value 

VC 

Set  voltage  to  a  constant  value 

VR 

Set  voltage  to  a  register  value 

RV 

Set  a  voltage  register  value 

RA 

Set  an  address  register  value 

TU 

Set  the  time  unit  register 

TR 

Set  the  tick  resolution  register 

LD 

Delay  a  constant  number  of  ticks 

LS 

Start  loop  with  a  constant  count 

LE 

End  loop,  decrement  count,  jump  to  start  if  count  >  0 

ET 

End  Tick 

RC 

Set  a  count  register  value 

LDR 

Delay  with  tick  count  loaded  from  count  register 

LSR 

Start  loop  with  count  loaded  from  count  register 

TRR 

Set  the  tick  resolution  from  a  count  register 

OIC 

Increment  a  COUNTf]  register  value 

ODC 

Decrement  a  COUNT[]  register  value 

ARC 

Set  address  from  ADDRESS[COUNT[x]] 

VRC 

Set  voltage  from  VOLTAGE[COUNT[x]] 

RSVD1 

No-op  (reserved) 

RSVD2 

No-op  (reserved) 

RSVD3 

No-op  (reserved) 

RSVD4 

No-op  (reserved) 

BNE 

Branch  if  COUNT[x]  !=  0 

BEQ 

Branch  if  COUNTjx]  ==  0 

ELSE 

Else  condition  for  if 

ENDIF 

Terminator  for  if 

RCC 

Copy  value  from  COUNT[b]  into  COUNT[a] 

RJMP 

Relative  jump  past  n  instructions 

ABI 

Address  increment  (block  mode) 

ABR 

Reset  address  (block  mode) 

VB 

Assign  channelVoltageQ  values  from  BiockVoitage[blocklndex][0- 
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channelCount][tactor] 

BCC 

Assign  channelCount  from  a  constant 

BSC 

Assign  blockSize  from  a  constant 

BIC 

Assign  blocklndex  from  a  constant 

MODE 

Assign  array  mode  from  a  constant 

BBC 

Assign  blockBufferCount  from  a  constant 

ABAC 

Assign  absolute  address  in  block  mode  from  a  constant 

BSR 

Assign  blockSize  from  a  count  register 

BCR 

Assign  blockCount  from  a  register 

BBR 

Assign  blockBufferCount  from  a  count  register 

BIR 

Assign  blockBufferlndex  from  a  count  register 

BCP 

Assign  BlockVoltage[ARGO][][]  values  to 
VOLTAGE[BlockVoltageRef[ARG1  ][][]] 

BCPR 

Assign  BlockVoltage[COUNT[ARGO]][][]  values  to 
VOLTAGE[BlockVoltageRef[COUNT[ARG1  ]][][] 

VBR 

Assign  channelVoltage[]  values  from 

VOLTAGE[BlockVoltageRef[blocklndex][0-channelCount][tactor]] 

WVUP 

Update  waveform  selections  from  BlockVoltage[bufidx][0][] 

PWM 

Start  PWM  with  freq=COUNT[ARGO]  duty=COUNT[ARG1] 

WVTS 

Activate  a  waveform  table  from  the  VM  table  list  (send  it  to  the 
FPGA) 

BNEO 

Branch  ARG2  steps  if  COUNT[ARGO+COUNT[X]]  !=  0 

ODCO 

Decrement  a  COUNT[ARGO+COUNT[ARG1]]  register  value 

BVR 

Set  a  BlockVoltageRef[ARGO][ARG1][COUNT[ARG2]]  register  to 
ARG3 

RCO 

Set  COUNT[ARGO+COUNT[ARG1]1  register  to  ARG2 

RCCO 

Set  COUNT[ARGO+COUNT[ARG1  ]]  = 
COUNT[ARG2+COUNT[ARG3H 

BEQO 

Branch  ARG2  steps  if  COUNT[ARGO+COUNT[XH  ==  0 

BVRR 

Set  blockVoltageRef[ARGO][ARG1][COUNT[ARG2]]  to 
COUNT[ARG3]  +  COUNT[ARG4] 
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APPENDIX  B  -  VIRTUAL  MACHINE  STATE  DEFINITION 


Data  Type 

Name 

Description 

UINT32 

pc 

Program  counter  (in  bytes) 

UINT32 

Ic 

Loop  counter  (current  index  into  loopcount,  increments 
when  a  new  loop  is  started,  decrements  when  a  loop 
completes) 

UINT32 

tu 

Time  unit  (defaults  to  microseconds) 

UINT32 

tr 

Tick  resolution  (#  of  ticks  in  „tu’  units  between  VM  ticks) 

UINT32 

et 

End  tick  (ends  a  run-to-completion  series) 

UINT32 

dr 

Delay  register  (holds  #  of  ticks  to  delay) 

UINT32R 

loopStart 

Stores  pc  when  a  loop  starts 

UINT32[] 

loopCount 

Loop  counts  (dimension  determines  max  #  of  nested  loops) 

UINT32R 

regV 

Voltage  registers 

UINT32R 

regA 

Address  registers 

UINT32R 

regC 

Count  registers 

UINT32 

ainc 

0  -  do  not  increment  address,  1  -  increment  address  (used 
for  row/column  IOD  block  mode) 

UINT32 

arst 

Address  reset  (0  do  nothing,  1  reset  tactor  address  to  0  on 
next  tick) 

UINT32 

dacval 

Current  DAC  value 

UINT32 

tactor 

Current  tactor  address 

UINT32 

arrayMode 

Current  array  configuration  mode 

0:  Discrete  tactor  scanning  mode 

1 :  Block  tactor  scanning  mode 

2:  Discrete  parallel  mode 

3:  Block  parallel  mode 

UINT32 

addressMode 

Current  tactor  addressing  mode 

UINT32 

voltageMode 

Current  tactor  voltage  mode 

0:  Single  DAC 

1:  Multiple  DAC 

2:  Simultaneous 

UINT32 

blockSize 

Number  of  factors  in  a  block  for  block  addressing  mode 

UINT32 

blockBufferCount 

Number  of  block  voltage  buffers  to  allocate  (corresponds  to 
number  of  full  “frames”  of  waveform  selections  for  each 
tactor) 

UINT32 

blockBufferlndex 

Index  into  the  blockVoltage  array  indicating  the  currently 
selected  block  voltage  buffer  to  present 

UINT32 

channelCount 

Number  of  channels/blocks  for  multi-DAC  mode 

UINT32R 

channelVoltage 

Current  DAC  value  for  each  channel  in  multi-DAC  mode 

UINT32[][][] 

blockVoltage 

Array  of  values  sized  by 

[blockBufferCount][channelCount][blockSize]  used  to 
indicate  which  voltages  should  be  applied  to  each  tactor 
and/or  which  waveforms  should  be  applied  to  each  tactor 

UINT32[] 

blockVoltageRef 

Array  of  indices  into  regV[]  that  can  be  used  to  copy  regV[] 
values  into  a  blockVoltage[] [][]  buffer 

UINT32 

fpgaFileCount 

Number  of  FPGA  files  (.VME)  loaded 

CHARflfl 

fpgaFilename 

Filenames  of  .VME  files  loaded 

UINT32 

waveformT  ableCount 

UINT32 

waveformT  ableCount 

Number  of  waveform  tables  that  are  loaded 

UINT8[][] 

waveform  DataTable 

Points  to  dynamically  allocated  waveform  table  arrays 

UINT32R 

waveformCounts 

Number  of  waveforms  in  each  table 

UINT32 

waveformSizes 

Sizes  of  each  waveform  in  bytes 
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APPENDIX  C  -  WGIO  PACKET  DEFINITIONS 


Command 

Arguments 

Description 

RDP 

Return  Data  Packet 

SPLS 

List  loaded  Stimulus  Programs 

SPSET 

Set  the  current  Stimulus  Program 

SPLD 

Load  a  Stimulus  Program 

SPDEL 

Delete  a  Stimulus  Program 

RASET 

Set  one  or  more  address  registers 

RVSET 

Set  one  or  more  voltage  registers 

RCSET 

Set  one  or  more  count  registers 

SPSTAT 

Get  device  status  packet 

SPGO 

Start/Resume  the  VM 

SPHALT 

Stop/Pause  the  VM 

PINGCFG 

Configure  PING  packets 

PING 

Ping  packet 

RESET 

Reset  the  VM 

CLEARSP 

Delete  all  Stimulus  Programs 

RAGET 

Get  the  value  of  an  address  register 

RVGET 

Get  the  value  of  a  voltage  register 

RCGET 

Get  the  value  of  a  count  register 

RBSET 

Set  one  or  more  block  voltage  buffer  registers  to  a  value 

BREFSET 

Set  one  or  more  BlockVoltageRef[][][]  registers  to  a 
value 

RBGET 

Request  a  packet  containing  the  value  of  a  block  voltage 
buffer  value 

CCONFIG 

Client  configuration  packet 

VMEFILE 

FPGA  .VME  file  packet 

WAVETBL 

Waveform  data  table  packet 

CLRDY 

Client  ready  packet 

RBREF 

Set  one  or  more  values  in  BlockVoltageRef[]  to  a  value 
from  the  VOLTAGE[]  array 

DEFINITIONS 

AFRL  Air  Force  Research  Laboratory 
API  Application  Program  Interface 

DARPA  Defense  Advanced  Research  Projects  Agency 

HD  High  Density 

HDA  High  Density  Array 

IRB  Internal  Review  Board 

Tactor  Electrode  on  stimulation  array,  comparable  to  pixel  on  visual  display 
TUNS  Tactical  Underwater  Navigation  Systems 
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